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School choice reforms comprise a broad category of policies aimed at improving public 
education through the introduction of market forces that expand customer choice and 
competition between schools. Here we summarize our four technical reports from the fourth and 
final year of research on the effects of a large statewide school voucher initiative, the Louisiana 
Scholarship Program (LSP), and draw the following conclusions: 


e Overall, participating in the LSP had a statistically significant negative impact on student 
English Language Arts (ELA) and math scores across most years of the evaluation, 
including the fourth year, and across most samples of students studied. 


e The achievement impacts of the LSP varied significantly based on the specific criterion- 
referenced test used and the accountability stakes attached to it. The negative test score 
impacts of the LSP were half as large in Year 1 when the test taken was the iLEAP, 
which is less aligned to the public school curriculum, than when it was the LEAP, which 
is more aligned. The impacts were not statistically significant in the third year of the 
evaluation, when no accountability stakes were assigned to the test scores. 


e The achievement effects of the LSP varied based on student characteristics. African 
American students experienced significantly less negative impacts of voucher usage after 
four years relative to non-African American students. 


e The achievement effects of the LSP varied based on the characteristics of the chosen 
schools. Students whose most-preferred schools were larger, charged higher tuition, and 
had longer school days experienced more favorable achievement impacts from 
participating in the LSP relative to their LSP peers who did not list such schools as their 
first choice. In some cases, test score impacts were positive and statistically significant 
for students choosing schools with these characteristics. 


e The effects of the LSP on college enrollment rates were neutral. Students who 
participated in the LSP in grades 7-12 starting in the fall of 2012 enrolled in college by 
2018 at arate of 60.0 percent compared to a rate of 59.5% for members of the 
experimental control group. The difference of 0.5 percentage points between the two 
groups is not statistically significant. The results were similar for students enrolling in 2- 
year or 4-year colleges. 


e Students applying to the LSP were disproportionately African American and low-income, 
compared to non-applicants. No consistent evidence indicates that the LSP is “cream 
skimming” or “pushing out” students based on their family social status or initial test 
scores. However, among LSP applicants, students with disabilities are less likely than 
students without disabilities to use a voucher initially. 


Combined with prior evidence, these results are informative about both the specific design of 
voucher and other choice policies as well as how the effects of choice vary across different 
outcomes and contexts. 


Introduction 


School choice has long been a subject of robust debate. Private school vouchers—programs 
providing public funds for students to attend K-12 private schools—tend to be the most 
contentious form of school choice. Over the past three years, our research team has released a 
series of reports examining how the LSP has affected key student and community conditions. In 
this brief, we summarize results from our final technical reports on the following questions: 


1. How did LSP voucher use affect student achievement after four years? 


2. Did the achievement effects of the LSP differ significantly based on the characteristics of 
the private schools preferred by students? 


3. How did LSP voucher use affect student educational attainment? 


4. What types of students applied to the program and used an LSP voucher initially and 
persistently? 


The Louisiana Scholarship Program 


Student performance on standardized tests in Louisiana has trailed national averages for decades. 
In an effort to turn things around, the state began offering students publicly financed vouchers to 
attend private schools in New Orleans in 2008. This pilot version of the LSP was expanded 
statewide in 2012. A total of 9,736 students applied to the program that year, with 5,296 
receiving vouchers. The program awarded 6,909 vouchers in 2017-18, a drop of 6% from its 
enrollment peak of 7,362 in 2015-16. 


The LSP is a statewide private school voucher program available to moderate- to low-income 
students in low-performing public schools. To qualify, children must have family incomes at or 
below 250% of the federal poverty line and either be entering kindergarten or be attending a 
public school that was graded C, D, or F for the prior school year by the state’s school 
accountability system. The majority of the program’s first year applicants applied from outside 
of New Orleans. This 2012-13 LSP applicant cohort is the subject of our evaluation. 


The voucher value is limited to 90% of the amount the state and local government provides in 
student funding to the local school system or the tuition charged by the student’s chosen private 
school, whichever is less. Average tuition at participating private schools ranges from $2,966 to 
$8,999, with a median of $4,925, compared to average per pupil spending of $8,500 in 
Louisiana’s public schools in the baseline year of 2012-13. 


To participate in the program, private schools must meet certain criteria related to: enrollment, 
financial practices, student mobility, and the health, safety and welfare of students. Participating 
schools are prohibited from being selective in their enrollment of voucher students and must 
administer the state’s accountability tests annually to voucher students in grades 3 through 8 and 
grade 10. 


Nearly 60% of applicants received vouchers for the 2012-13 school year. Of these recipients, 
86% used their vouchers to enroll in private schools in the first quarter of 2012-13. 


Roughly 87% of the applicants are African American, with 8% white and 3% Hispanic. Prior to 
applying to the LSP, students performed below the state average in ELA, math, science, and 
social studies by around 20 percentile points on the state accountability test. Applicants to the 
program in 2012-13 were concentrated in the earlier grades, with one-third entering Kindergarten 
through third grade. 


Louisiana offers three private school choice programs in addition to the LSP. First, the state 
offers taxpayers a tax deduction of up to $5,000 per child for education expenses, including 
private school tuition. Over 100,000 Louisianans received the deduction in 2012. Second, 1,703 
Louisiana students received a scholarship from a privately-funded School Tuition Organization 
to attend private school through the state’s Tuition Donation Rebate Program in 2017-18. 
Finally, the state offers a separate voucher program for students with disabilities, the School 
Choice Program for Certain Students with Exceptionalities (SCPCSE). Launched in 2011, the 
SCPCSE is intended to expand the educational options for students with unique educational 
needs. SCPCSE vouchers are restricted to the lesser of the private school’s tuition or 50 percent 
of the state funds that would have been spent on the student, which means the vouchers can be 
worth less than an LSP voucher depending on the severity of a student’s disability. In 2017-18, 
the average SCPCSE voucher was worth $2,500. Eligibility is limited to parishes (a.k.a. 
counties) with at least 190,000 residents. The program only enrolled 394 students in the 2017-18 
school year. 


We are not able to evaluate the effects of these other private school choice programs on student 
achievement because student achievement data are not collected for their participants and two of 
them are small. Our evaluation is therefore limited to the LSP and does not capture the effects of 
the state’s subsidized private school choice in general. 


Prior findings 


One of the themes of this brief is that the voucher landscape and research are quickly evolving. 
In a series of reports we released in 2016 and 2017, we focused on earlier test scores results, the 
effect of the program on students with disabilities, private school participation in the program, as 
well as results for non-academic outcomes, competitive pressures on public schools, and racial 
integration. From that work, we drew the following conclusions: 


e Overall, participating in the LSP has no statistically significant impact on student English 
Language Arts (ELA) or math scores after using an LSP voucher for three years. 
Achievement impacts varied over time. The effects were particularly negative after the 
first year and were slightly less negative after two years. 


e The subgroup of students who were lower achieving before applying to the program did 
show significant gains in ELA after three years of voucher usage. Students applying to 
lower grades demonstrated significant losses in math. 


e Students participating in the LSP were less likely than their control group peers to be 
identified as having a disability after two or three years and were more likely to be de- 
identified as having a disability after two years in the program. 


e The private schools that chose to participate in the LSP were disproportionately Catholic, 
had low tuitions that were close to the voucher amount, and served a high percentage of 


minority students. 


e We found no evidence that the LSP impacted students’ non-academic skills, such as 
conscientiousness and grit, due in part to unreliable measures of these traits. 


e Achievement of students in Louisiana public schools facing increased competitive 
pressures from the LSP was either unaffected or modestly improved as a result of the 
program’s statewide expansion in 2012-13. 


e The majority of LSP transfers improved integration in students’ former public schools; 
however, LSP transfers slightly worsened integration in new private schools. The net 
effect of the program was positive, as more transfers helped than harmed integration, 
especially in districts subject to court orders for prior racial segregation. 


e The majority of public school districts in Louisiana would be squeezed, fiscally, if the 
LSP was ended and the program’s students returned to traditional public schools. 


Vouchers and other forms of school choice raise many questions and require comprehensive 
program evaluations. The research that follows builds on these earlier studies, providing one of 
the most comprehensive evaluations of any voucher program in the country. 


These prior LSP reports and our latest set of studies all can be found at 
https://scdp.uark.edu/louisiana-scholarship-program-evaluation/. 


How did LSP voucher use affect student achievement after four years? 


The first report in this series, by Jonathan Mills and Patrick Wolf, examines how LSP voucher 
use affects student achievement. Achievement plays an important role in how the Louisiana 
Department of Education monitors the LSP’s success, as private schools receive sanctions for 
continually low performance. Thus, we follow in a long tradition of evaluating the effect of 
school voucher programs in part by analyzing student test scores. 


We determine the impact of LSP voucher use on student achievement by comparing students 
who received vouchers through randomized lotteries. The LSP was oversubscribed in the first 
year of the program and used a matching algorithm to allocate open seats in private schools to 
students. When LSP applicants exceeded the number of seats available in a given school, the 
program awarded voucher placements to that specific school by lottery. Our analysis focuses on 
this subset of eligible applicants whose voucher receipt was determined randomly so that any 
differences in outcomes between LSP awardees and non-awardees can be attributed to the 
program. 


We examine how the LSP effects vary over time for two samples of students: a sample restricted 
to students with baseline test scores (BA) and a larger sample that is not limited to students with 
baseline test scores (NBA). Our prior research indicates large negative impacts on ELA and math 
in the first year of participation that appear to diminish somewhat by Year 2 and are not 
statistically significant by Year 3. Figures 1 and 2 present regression estimates for ELA and math 
LSP effects for Years | through 4 for consistent samples of students in the study. Figure 1 
presents results for students in the BA Sample and Figure 2 focuses on students in the larger and 
broader NBA Sample. 
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Figure 1. Estimated effect of ever using an LSP Voucher over time for BA sample. 

Figure presents point estimates from fully specified models for 2011-12 (baseline) through 2015-16 for ELA and 
math. Results are presented for a consistent sample of students with Spring 2016 outcome data. ELA and math 
results are based on student achievement on the Louisiana state assessments (LAA) in 2011-12 through 2013-14, 
PARCC assessments in 2014-15, and LAA in 2015-16. Dashed lines represent 90% confidence intervals for the 
performance averages. 
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Figure 2. Estimated effect of ever using an LSP Voucher over time for NBA sample. 

Figure presents point estimates from fully specified models for 2013-14 through 2015-16 for ELA and math. Results 
are presented for a consistent sample of students with Spring 2016 outcome data. ELA and math results are based on 
student achievement on the Louisiana state assessments (LAA) in 2012-13 through 2013-14, PARCC assessments in 
2014-15, and LAA in 2015-16. Dashed lines represent 90% confidence intervals for the performance averages. 


Consistent with our prior work, we observe large declines in ELA and math performance in the 
first year of voucher usage that become less negative in Years 2 and 3. Effects are generally 
worse in math than in ELA. By Year 4, however, we observe a direction reversal for the voucher 
usage effect estimates. Effects are slightly more negative in magnitude in Year 4 relative to Year 
3 across all tests and samples and are statistically significant for math in both samples and ELA 
in the NBA sample only. 


In general, we do not observe consistent evidence that LSP voucher usage achievement effects 
differ by gender. In contrast, we do observe evidence suggesting effects were experienced 
differently by students of different racial backgrounds (Figure 3). The achievement effects of the 
LSP are generally less negative for African American students relative to students of other races 
and ethnicities. Previous voucher evaluations have reported similar evidence of achievement 
effects being relatively more favorable for African Americans. Year 4 is the first time we have 
observed such evidence for the LSP. 
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Figure 3. Differential effects of LSP usage by race for NBA sample. 

Figure presents separate estimates of the impact of ever using an LSP voucher to attend a private school between 
2012-13 and 2015-16 for African American students and students of other races and ethnicities for ELA and math. 
Models are fully specified. *** - p<.01, ** - p<.05, * - p<0.10. 


Did the achievement effects of the LSP differ significantly based on the 
characteristics of the private schools preferred by students? 


Some prior studies have found that the test score effects of school choice programs can vary 
based on key features of the specific schools that are chosen. Our study builds on this previous 
work with an exploratory analysis of the variation in LSP achievement effects across 13 school 


characteristics in the first four years of the program. By Year 4, our analytic method was too 
imprecise to yield any clear results, so we concentrate on results from the first three years. 


In general, we do not observe effect heterogeneity across school characteristics, though we find 
evidence suggesting students who preferred larger schools, schools with higher tuition, and 
schools with longer school days experienced more favorable impacts from participating in the 
LSP relative to their peers who did not prefer such schools (Figure 4). Students whose first- 
choice private school was in the top third of the distribution for total K-12 enrollment actually 
experienced a positive LSP math impact in 2014-15. 
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Figure 4. Variation in LSP math impacts across terciles of school tuition charged, total school 
enrollment, and school day hours, 2012-13 through 2014-15. 


Notes. Figure presents point estimates from regression models for 2012-13 through 2014-15 for standardized math 
impacts of ever using an LSP voucher placement. Estimates in 2014-15 for tuition and school day hours, as well as 
2015-16 generally, failed a reliability test and therefore are not presented. *** p<.01, ** p<.05, * p<0.10. 


Several private school characteristics did not appear to influence the achievement impacts of the 
LSP. These included the school’s religious status, coeducational status, urbanicity, student- 
teacher ratio, length of school year, total instructional hours, and racial and ethnic composition. 


How did LSP voucher use affect student educational attainment? 


How long a student remains engaged in school greatly influences his or her later life outcomes. 
As aresult, researchers increasingly study the impacts of education programs on student rates of 
high school graduation and college enrollment, persistence, and completion. Too few students in 
our experimental sample are old enough to have graduated from college for us to examine that 
important outcome. However, over 1000 students who faced LSP lotteries are old enough to have 
enrolled in college. Did the negative test score effects of the LSP decrease their rates of college- 
going? 


We find that the LSP had no statistically significant effect on college entrance for students. As 
described in Figure 5, students who received an LSP voucher were more likely to enter college 
by 0.5 percentage points compared to students who did not win a voucher lottery. The estimated 
effect was small and statistically insignificant, with large standard errors surrounding a near-zero 
average effect. 
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Figure 5. Regression-adjusted college enrollment rates for students who ever used an LSP 


voucher and students in the control group. 
Enrollment rate is for enrolling in any two-year or four-year institution of higher education at any time between 
2013 and 2018. Regression adjusted for student and family demographic characteristics. 


The majority of students in the analysis enrolled in college, with 59.5% of control students and 
60.0% of treatment students having entered college by 2018. That college-going rate is 
particularly high given that students who applied for the program came from economically 
disadvantaged backgrounds. These higher than average college enrollment rates could be a result 
of other efforts Louisiana has made to expand access to college, such as the Taylor Opportunity 


Program for Students (TOPS) scholarship. 


We also estimated the likelihood of participating students entering two-year or four-year 
institutions. We found that the LSP had no significant effect on the rate at which students chose a 
four-year over a two-year post-secondary institution. For four-year colleges, LSP students 
enrolled at a rate that was 2 percentage points higher than control students, but the difference 
was not statistically significant. 


What types of students applied to the program and used an LSP voucher 
initially and persistently? 


Private school choice programs often are accused of failing to serve disadvantaged students. 
Instead, critics claim that participating private schools “skim the cream off the top” by admitting 
only the best students and “push out” students who are the most difficult to teach. This study 
tested student selection hypotheses in the context of the Louisiana Scholarship Program (LSP). 


Applicants to the program disproportionately had low test scores and came from disadvantaged 
populations (Table 1). We found that LSP applicants were less advantaged than their public 
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school peers regarding their family social status and initial test scores. No consistent evidence 
indicated that the LSP is “cream skimming” or “pushing out” students based on their family 
social status or initial test scores. However, students with disabilities were less likely than 
students without disabilities to use a voucher initially. Students in districts with lower per-pupil 
spending and fewer charter school options also were more likely to use an LSP voucher initially 
and persistently. 


Table 1. Student, private school, and public school district characteristics that influenced 
LSP application and voucher use 

Characteristic Application | Initial | Persistent | Year 3 User v. 
Use Use Non-applicant 


Student 

Low-income 

African American 
Hispanic 

Early grades 

Lower initial scores 
No IEP 

NOLA Pilot 
Attended magnet 

Did not attend charter 
Female + + 
Private School 
Lower minority enrollment + + 
Higher tuition + 
Shorter distance from home + + 
Not first-choice of student 
Public School District 
Lower per-pupil spending + + 
Fewer charters + 


+ 
+ 


+[+]+]+]+ 


+]t]+]+ 


Students with lower initial test scores were more likely to remain in the LSP than were students 
with higher initial test scores. Students who were placed in LSP private schools farther from their 
homes or that served a larger minority population were more likely to leave their LSP schools 
than LSP students placed in schools closer to their home or that served smaller minority 
populations. Finally, after all of these patterns of student selection played out, the LSP students 
still using vouchers after three years were more likely to have entered the program in the early 
grades, have a low family income, be African American, and be female than the population of 
non-applicants to the program. 


What do these results mean for private school choice? 


Before considering what these impacts mean for private school choice, it is important to realize 
the Louisiana Scholarship Program’s uniqueness. The LSP is one of only three statewide private 
school choice programs that combines parental choice of school with results-based 
accountability driven by scores on the state test. The Indiana Scholarship Program and the 
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Wisconsin Parental Choice Program are the other such programs. Among these programs, the 
LSP is the only statewide voucher program that combines that state testing requirement with an 
open admissions mandate for the private schools. Therefore, the results from this evaluation may 
not apply to the 55 other private school choice programs across the country that differ from the 
LSP regarding that combination of design features. 


Moreover, it is important to consider the uniqueness of our analytical sample when considering 
the applicability of our findings. Our reports were limited to the experiences of the initial cohort 
of LSP applicants for the 2012-13 school year. Subsequent cohorts of program participants, all of 
which were much smaller than the first cohort, may have had different experiences in the LSP 
than the original group. 


Some evidence from our reports suggests that the curricular alignment of the state test and the 
stakes attached to it may have influenced the pattern of LSP test score effects we found. The 
negative achievement impacts of the LSP were larger for grades administered the LEAP test, 
which is more aligned to the public school curriculum, than for grades administered the iLEAP 
test, which is less aligned. The negative impacts also were larger in Years 1, 2 and 4, when the 
state accountability test was high-stakes for the public schools, but smaller (and even 
insignificant in one of our samples) in Year 3, when a new state test did not count towards 
accountability ratings. 


Moreover, in spite of scoring lower than their control group peers, especially in math, students 
who participated in the LSP were accepted to and enrolled in college at a rate that was 
statistically similar to the students who lost their LSP placement lottery. One possible 
explanation for those divergent results is that private schools in the LSP are teaching students 
content and skills that help them get accepted to college but are not measured by the state test. 


While the LSP appears to have negatively impacted student scores on the state test and had little 
impact on the likelihood of a student attending college, there is more to this story. The transfer of 
almost 5,000 students out of public schools and into private schools in the first year of the 
program left those public schools better racially integrated. African American participants in the 
LSP experienced much smaller achievement losses from the program in Year 4 compared to 
students of other races and ethnicities, though they still experienced losses. Students whose first- 
choice private schools were relatively high in school enrollment, tuition charged, and the length 
of the school day scored as well on the state test as their control group peers in most years and 
subjects, even demonstrating a positive LSP impact in math in Year 3 for students in higher- 
enrollment LSP schools. If we think that school enrollment is a reasonable measure of parent 
demand, and tuition price is a rough measure of school quality, the students who won lotteries to 
higher-demand and higher-quality private schools in the LSP did not experience significant 
achievement losses, on average. 


The LSP was designed to provide private school options for disadvantaged families. The 
evidence indicates it accomplished that goal. Program applicants were disadvantaged relative to 
the average Louisiana student. Their patterns of using an LSP voucher placement initially and 
persistently, if awarded one, provide no consistent evidence of cream-skimming or push-out on 
the part of the private schools. While students with disabilities were less likely to use a voucher 
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if offered one, students with lower initial test scores were more likely to continue in the program 
through three years. Students also remained in the program longer if their district public schools 
received less funding and there were fewer public charter school options nearby. 


Finally, it is important to remember that Louisiana is a unique context in which to launch a 
statewide private school voucher program. Its Elementary and Secondary School Tuition 
Deduction policy permits parents to deduct up to $5,000 per child off their state taxable income 
for tuition and fees paid to private K-12 schools. Over 100,000 taxpayers benefited from the 
deduction in 2012. That tax policy makes it easier for Louisiana families to afford to send their 
children to private schools, and the schools themselves face no additional regulations as a result. 
It might be especially challenging to persuade many private schools to participate in a means- 
tested voucher program when their tuition-paying families already are being aided through such 
a generous tax deduction. 


These reports conclude our longitudinal evaluation of the Louisiana Scholarship Program. We 
have uncovered a mix of negative, positive, and null effects of this unique private school choice 
program on participating students, non-participating students, and schools. We appreciated the 
opportunity to bring this evidence to bear on current and future public debates about private 
school choice in the Pelican State. 
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